Adaptive optimization of a content item using continuously trained machine learning models

ABSTRACT

A processor receives requests for content items and identifies a first subset of machine learning (ML) models that satisfy a reliability criterion and a second subset of ML models that fail to satisfy the reliability criterion, wherein each ML model is associated with a respective content template and is trained to output a probability that a target associated with an input set of characteristics would perform a target action responsive to being presented with a content item generated based on the respective associated content template. The processing logic assigns each request to either a first group or a second group based on a ratio of a number of ML models in the first subset to a number of ML models in the second subset. For each request in the first group, the processor generates a content item based on a content template associated with the first subset.

RELATED APPLICATIONS

This patent application is a continuation application of U.S. patentapplication Ser. No. 16/831,627, filed on Mar. 26, 2020, and claims thebenefit under 35 U.S.C. § 119(e) of U.S. Provisional Application No.62/956,935, filed Jan. 3, 2020, the entire contents of which are herebyincorporated by reference herein.

TECHNICAL FIELD

Embodiments of the present disclosure relate, in general, to optimizingcontent, and in particular to optimizing content of web components usingmachine learning.

BACKGROUND

Web contents can contain various components, such as a text, an audio, avideo, and an image. Such components can be combined in many differentways. Systems that generate web content from these numerous combinationsmay suffer from an effect of combinatorial explosion. Combinatorialexplosion describes the effect of functions that grow rapidly (e.g.,exponentially) as a result of combinatorial considerations.Combinatorial explosion can pose a problem in computing, as there may beinsufficient resources (e.g., insufficient memory, processing power,time, etc.) to test every possible unique combination of componentvalues and determine an optimal combination of component values based onthe testing.

BRIEF DESCRIPTION OF THE DRAWINGS

Aspects and implementations of the present disclosure will be understoodmore fully from the detailed description given below and from theaccompanying drawings of various aspects and implementations of thedisclosure, which, however, should not be taken to limit the disclosureto the specific aspects or implementations, but are for explanation andunderstanding only.

FIG. 1 illustrates an example of a system architecture forimplementations of the present disclosure.

FIG. 2 depicts an example of a content managing component, in accordancewith some embodiments of the disclosure.

FIG. 3A illustrates an example template of a content item in accordancewith some embodiments of the present disclosure.

FIG. 3B illustrates another example template of a content item inaccordance with some embodiments of the present disclosure.

FIG. 3C illustrates an example content item generated from a template ofFIG. 3A in accordance with some embodiments of the present disclosure.

FIG. 4 depicts a flow diagram of a method for generating a content item,in accordance with some embodiments of the present disclosure.

FIG. 5A depicts a flow diagram of a method for generating a content itembased on a selected content template, in accordance with someembodiments of the present disclosure.

FIG. 5B depicts a flow diagram of a method that is continued from themethod of FIG. 5A for generating a content item based on a selectedcontent template, in accordance with some embodiments of the presentdisclosure.

FIG. 6 depicts a block diagram of an example computing system operatingin accordance with one or more aspects of the present disclosure.

DETAILED DESCRIPTION

Embodiments are directed to methods and systems for using machinelearning models to generate web content having an optimal combination ofcontent components within a reduced amount of time by using minimalresources. A web content can correspond to any form of informationprovided via a web page. Within a web content, there may be differenttypes of audiovisual components (e.g., a title, a call to action, animage, a video, an audio, a background image, a message, etc.) laid outin various ways on a web page. The optimal combination of thesecomponents for web content may achieve a defined result (e.g., aresponse action from a target audience of the content that can bemeasured by a key performance indicator (KPI)). However, to identify theoptimal combination, millions to billions of different possiblecombinations of content components should be tested for thousands tomillions of different target audiences. Individual testing of eachcombination for each target audience would take enormous processingresources and time. Embodiments described herein provide an adaptiveoptimization technique for a web content by initially using randomcombinations of content, and using machine learning models that detectedstrong data patterns in key performance indicators associated with thecombinations of content and at the same time gradually training otherpremature machine learning models for which data patterns have not yetemerged. Embodiments save the processing time and resources in thegeneration and application of machine learning models that can determinecontent combinations to output based on target audiences.

Embodiments are discussed with reference to generating and deliveringweb content. However, it should be understood that the embodimentsdiscussed herein also apply to other systems in which there are manydifferent combinations of components that are possible for a largevariety of targets. For example, embodiments may determine email contentto deliver to a target, a text message (e.g., an MMS message) to deliverto a target, a physical mailer to send to a target, a video to output toa television of a target, and so on. Accordingly, it should beunderstood that discussions associated with web content herein alsoapply to many other different scenarios, such as email content, messagecontent, mailer content, video content, and so on.

FIG. 1 illustrates an example of a system architecture 100 forimplementations of the present disclosure. The system architecture 100includes a content generating server 112 and content providing servers118A-118Z connected via a network 130. Moreover, the system architecture100 includes client devices 120A-120Z, 120ZA-120ZZ connected to thecontent providing servers 118A-118Z connected via networks 140A-140Z.

The network 130 and the networks 140A-140Z may be public networks (e.g.,the Internet), private networks (e.g., a local area network (LAN) orwide area network (WAN)), or a combination thereof. The network 130and/or the networks 140A-140Z may include a wireless infrastructure,which may be provided by one or more wireless communications systems,such as Wi-Fi hotspot connected with the network 130 and/or the networks140A-140Z and/or a wireless carrier system that can be implemented usingvarious data processing equipment, communication towers, etc.Additionally or alternatively, the network 130 and/or the networks140A-140Z may include a wired infrastructure (e.g., Ethernet). In someembodiments, the network 130 and the networks 140A-140Z can be a singlenetwork. In some other embodiments, the networks 140A-140Z can form asingle network.

The content generating server 112 may be represented by one or morephysical machines (e.g., server machines, desktop computers, etc.) thatinclude one or more processing devices communicatively coupled to memorydevices and input/output (I/O) devices. The content generating server112 may receive requests for content items from the content providingservers 118A-118Z. A content item can correspond to an item visually oraurally presented on a web page supported by the content providingservers 118A-118Z. Examples of a content item can include a personalizedadvertisement, media (a video or music) or consumer electronicsrecommendation, a landing page, a check out page, a packaging or a coverdesign, an email or a text message content, message of a physical mail,a video for a television, and so on The content generating server 112may process the received requests and generate content items for therequests, as will be described below with respect to FIGS. 4, 5A, and5B. In one embodiment, the content generating server 112 may provide thegenerated content items to the content providing servers 118A-118Z forpresentation to the client devices 120A-120Z, 120ZA-120ZZ.

The content providing servers 118A-118Z may be represented by one ormore physical machines (e.g., server machines, desktop computers, etc.)that include one or more processing devices communicatively coupled tomemory devices and input/output (I/O) devices. The content providingservers 118A-118Z may provide a web page or any other medium thatcontains various contents (e.g., a personalized advertisement, media (avideo or music) or consumer electronics recommendation, a landing page,a check out page, a packaging or a cover design, an email or a textmessage content, message of a physical mail, a video for a television)to the client devices 120A-120Z, 120ZA-120ZZ. The content providingservers 118A-118Z may also gather characteristics about target audiencesof the client devices 120A-120Z, 120ZA-120ZZ. Such characteristics caninclude demographic information (such as, an age or a gender),contextual information (such as, a brand of the client devices120A-120Z, 120ZA-120ZZ, an operating system of the client devices120A-120Z, 120ZA-120ZZ, a time zone, a geographic location), historical(or user behavioral) features (such as, a number of impressions, timesince the last impression, a number of clicks).

The content providing servers 118A-118Z may receive requests for webcontents from the client devices 120A-120Z, 120ZA-120ZZ. In response,the content providing servers 118A-118Z may send requests for content tothe content generating server 112. Alternatively, or additionally, thecontent providing servers 118A-118Z may request content from the contentgenerating server 112 without first receiving requests for content fromthe client devices 120A-Z, 120ZA-ZZ. The content providing servers118A-118Z can also provide the corresponding characteristics associatedwith the client devices 120A-Z, 120ZA-ZZ (which may includecharacteristics of users of such client devices) along with the requeststo the content generating server 112.

Subsequently, the content providing servers 118A-118Z may receive therequests for content from the content generating server 112. The contentproviding servers 118A-118Z may generate content items based on therequests and provide the content items to the client devices 120A-120Z,120ZA-120ZZ. Different content items may be provided to different clientdevices 120A-Z, 120ZA-ZZ based at least in part on the characteristicsassociated with those different client devices 120A-Z, 120ZA-ZZ and orrespective users of the client devices 120A-Z, 120ZA-ZZ. Responsive toreceiving the content items, client devices 120A-120Z, 120ZA-120ZZ mayor may not receive user interaction with the content items, which may beassociated with KPIs. Furthermore, the content providing servers118A-118Z may receive any responses to the presented content items fromthe client devices 120A-120Z, 120ZA-120ZZ. The content providing servers118A-118Z may subsequently provide the responses to the contentgenerating server 112.

The client devices 120A-120Z may include one or more processing devicescommunicatively coupled to memory devices and I/O devices. The clientdevices 120A-120Z may be desktop computers, laptop computers, tabletcomputers, mobile phones (e.g., smartphones), or any suitable computingdevice. As discussed above, the client devices 120A-120Z may eachinclude a web browser and/or a client application (e.g., a mobileapplication or a desktop application) for viewing contents (includingcontent items) provided by the content providing servers 118A-118Z viauser interfaces 124A-124Z, 124ZA-124ZZ supported by a web browser and/ora client application.

FIG. 2 depicts an example of a content managing component 116, inaccordance with some embodiments of the disclosure. The content managingcomponent 116 can include a communication module 210, a contentdetermining module 220, machine learning models 230, and a trainingengine 240. The components can be combined together or separated intofurther components, according to particular implementations. It shouldbe noted that in some implementations, various components of the contentmanaging component 116 may run on separate machines.

The communication module 210 can communicate with the content providingservers 118A-118Z to receive requests for a content item, correspondingsets of characteristics to the requests, and any responses (e.g., aclick on the content item, a viewing of the content item, a conversion,after having seen or clicked the content item, purchasing an itempresented in the content item or any other type of conversion) to thecontent item. The communication module 210 can also provide therequested content item to the content providing servers 118A-118Z.

The training engine 240 can train the machine learning models 230. Inone embodiment, the training engine can periodically train the machinelearning models 230 in multiple phases, thereby increasing modelaccuracy as more training data accumulates. The training engine 240 cantrain the machine learning models 230 to solve a probability or scoreestimation problem—that is, whether a content item associated with arespective machine learning model 230 is most likely to achieve aresponse action from a target (the achievement can be measured by a keyperformance indicator (KPI)). This task class deals with ranking tasks,therefore both probability and score can be applied for differentmodels, the only requirement is the presence of a common scale forcomparison. The training engine 240 may find patterns in training data(including training input and training output) that map the traininginput to the target output (i.e., the answer to be predicted) andprovide the machine learning model 230 that captures these patternsunder supervised learning. Accordingly, the trained machine learningmodels 230 can each predict a probability of a target audience having arespective set of input characteristics performing a target action (suchas, a click) in response to being presented with a respective contentitem.

Some of the machine learning models 230 may be associated with adifferent content templates, and may process an input set ofcharacteristics to determine a probability of a key performanceindicator being achieved if a content item based on the associatedcontent template is presented to a target. The outputs of multiplemachine learning models 230 may be ranked, and an optimal (based oncurrent information) content template may be determined. The optimalcontent template may be for the content item specific to a target withthe input set of characteristics.

Other machine learning models 230 may be associated with a differentcontent item component of a content template, and may process an inputset of characteristics to determine a probability of a key performanceindicator being achieved if the content item including the associatedcontent item component is presented to a target. The outputs of multiplemachine learning models 230 may be compared, and an optimal (based oncurrent information) set of content item components may be determined.The Optimal set of content item components together may form a contentitem in accordance with an associated content template, specific to atarget with the input set of characteristics. More details abouttraining the machine learning models 230 and using the trained machinelearning models 230 are described below with respect to FIGS. 4, 5A, and5B.

The content determining module 220 can generate a content item by usingthe trained machine learning models 230 and a set of characteristicsreceived from the communication module 210, as will be further describedwith respect to FIGS. 4, 5A, and 5B.

FIG. 3A illustrates an example template 301 of a content item inaccordance with some embodiments of the present disclosure. The template301 can include multiple components 302A to 302C. For example,components 302A to 302C can be skins, backgrounds, titles, texts,call-to-actions, etc. Each component 302A to 302C may be associated oneor more elements. That is, each component 302A to 302C can havedifferent options such as different colors of the background component.If, for example, there are 100 options for the component 302A, 100options for component 302B and 100 options for component 302C, thenthere are 100×100×100=1,000,000 different content items that can becreated for this simple template 301. More complex templates may have anincreased number of components, which may further increase the totalnumber of combinations, and thus the number of different unique contentitems that can be generated from the template.

FIG. 3B illustrates another example template 303 of a content item inaccordance with some embodiments of the present disclosure. Although thetemplate 303 may be a template for the same content item as that of thetemplate 301, the template 303 can include a different number ofcomponents, as well as different types of components. In the illustratedexample, template 303 includes components 304A to 304D.

FIG. 3C illustrates an example content item 350 generated from template301 of FIG. 3A in accordance with some embodiments of the presentdisclosure. As illustrated, the content item 350 includes threecomponents: a headline component, an image component, and a call toaction component in accordance with the template 301 of FIG. 3A. Basedon the methods described below with respect to FIGS. 4, 5A, and 5B, thefirst headline element out of four elements is selected to present thetext, “Skateboard Artist” for the headline component. As for the imagecomponent, a first image out of three image elements is selected todisplay a man with a skateboard. Lastly, for a call to action component,the first button from four elements is selected to implement theinteractive button, “Get Started Today.”

The present disclosure describes adaptive optimization of a content itemusing continuously trained machine learning models. That is, the systemsand methods described herein adaptively optimizes content items bytraining machine learning models in multiple phases.

At an initial phase of generating content items where machine learningmodels are not yet trained because there is no (or not enough) data onKPIs or responses to any of the templates or elements, a content itemcan be randomly or pseudo-randomly generated. For example, such acontent item can include a random or pseudo-random number and types ofcontent components and corresponding elements may be generated. Onceenough initial data (responses from target audiences of content items)is gathered, machine learning models for each template and for eachelement of each component in a template can be generated and trained.

In one embodiment, machine learning models can be trained based on alimited number of training data, such as a couple of thousands sets oftraining data or the training data collected over a couple of days basedon responses to the randomly or pseudo-randomly generated content items.The processing device can include a training engine that is capable oftraining a machine learning model. The training engine may find patternsin training data (including training input (sometime, referred to asfeatures) and training output (sometimes, referred to as a target labelor target output)) that map the training input to the training output(i.e., the answer to be predicted) and provide the machine learningmodel that captures these patterns under supervised learning. Such amachine learning model can correspond to a model artifact that iscreated by the training engine that uses training data (e.g., traininginputs and corresponding training outputs (i.e., correct answers forrespective training inputs)). The machine learning model may be composedof, e.g., a single level of linear or non-linear operations based onlogistic regression or gradient boosting technique.

As an example, a machine learning model can be trained based on traininginputs including multiple sets of characteristics describing a target(e.g., a target audience of the content item), and correspondingtraining outputs indicating whether or not target actions (e.g., a clickon the content item, a viewing of the content item, a conversion, afterhaving seen or clicked the content item, purchasing an item presented inthe content item or any other type of conversion) are performed by thetargets responsive to being presented with content items based on theassociated content template. As a result of the training, the trainedmachine learning model can provide a probability that a target having aset of characteristics would perform a target action in response tobeing presented with a content item generated based on the respectivecontent template associated with the respective trained machine learningmodel.

After the initial training, a processing device of the presentdisclosure can group the machine learning models into the first subsetor the second subset based on a reliability criterion, as well as anycorrelation to KPIs. For example, the processing device can group thetrained machine learning models associated with content templates intotwo subsets. The first subset can include machine learning models thatsatisfy the reliability criterion (i.e., that are determined to besufficiently reliable, stable, or accurate). On the other hand, thesecond subset can include machine learning models that does not yetsatisfy the reliability criterion (i.e., that are determined to be notsufficiently reliable, stable, or accurate yet — that is, more trainingdata is needed). The processing device can similarly group the trainedmachine learning models associated with elements of content componentsin a content template.

In one implementation, the processing device can utilize a reliabilitycriterion that is associated with at least one of a mean or a standarddeviation of an area under the ROC (receiver operating characteristic)curve that is generated using one or more sets of validation data with arespective trained machine learning model. The validation data caninclude validation input data as a set of characteristics associatedwith a target and validation output data as an indication of whether ornot a target action was performed. For example, the processing devicecan use a multi-fold cross-validation technique. Accordingly, for eachfold (e.g., each set of validation data), the processing device candetermine an area under ROC curve (AUC). Based on the AUC, theprocessing device can determine a mean and standard deviation of theAUC. Subsequently, the processing device can determine that a trainedmachine learning model satisfies a reliability criterion in response todetermining that a) a difference between the mean and standard deviationof the AUC is greater than a threshold value (e.g., 0.5) and b) thestandard deviation of the AUC is equal to or less than a threshold value(e.g., 0.1). Accordingly, any trained machine learning models thatsatisfies the reliability criterion is categorized as the first subsetand the trained machine learning models that does not satisfy thereliability criterion is categorized as the second subset. The nextphase of generating content items using continuously trained machinelearning models are described below with respect to FIGS. 4, 5A, and 5B.

FIG. 4 depicts a flow diagram of a method 400 for generating a contentitem, in accordance with some embodiments of the present disclosure. Themethod 400 may be performed by the content managing component 116 ofFIG. 1 . The method 400 may be executed by one or more processingdevices of the server 112. The method 400 may be carried out for eachcontent providing server 118A-118Z to be presented to client devices120A-120Z, 120ZA-120ZZ.

For simplicity of explanation, the method 400 of this disclosure isdepicted and described as a series of acts. However, acts in accordancewith this disclosure can occur in various orders and/or concurrently,and with other acts not presented and described herein. Furthermore, notall illustrated acts may be required to implement the method 400 inaccordance with the disclosed subject matter. In addition, those skilledin the art will understand and appreciate that the method 400 couldalternatively be represented as a series of interrelated states via astate diagram or events. Additionally, it should be appreciated that themethod 400 disclosed in this specification are capable of being storedon an article of manufacture (e.g., a computer program accessible fromany computer-readable device or storage media) to facilitatetransporting and transferring such method to computing devices.

At operation 410, the processing device receives requests for contentitems. The processing device can receive such requests from contentproviding servers (the content providing server 118A-118Z). The contentitems can later be included, by the content providing servers, in a webpage to be loaded by a web browser of the client devices (the clientdevices 120A-120Z, 120ZA-120ZZ). Examples of a content item can includea personalized advertisement, media (a video or music) or consumerelectronics recommendation, a landing page, a check out page, apackaging or a cover design, etc.

Each request received at operation 410 can be associated with arespective target (e.g., a target audience of the content item)including a set of characteristics. That is, the target can berepresented by a set of characteristics such as demographiccharacteristics (such as age or gender), contextual characteristics(such as device brand, operating system, user time, geographiclocation), historical (or user behavioral) characteristics (such as anumber of impressions, time since the last impression, number ofclicks). In one embodiment, each request can include data representingsuch a set of characteristics.

At operation 420, the processing device identifies a first subset oftrained machine learning models that satisfy a reliability criterion anda second subset of the plurality of trained machine learning models thatfail to satisfy the reliability criterion. Refer to the above for moredetails about the reliability criterion. In one implementation, eachtrained machine learning model is associated with a content template(out of multiple content templates available for the requested contentitem). That is, each trained machine learning model can providepredictions of how likely it is for a target (e.g., a target audience ofthe content item) to respond to a content item generated using theassociated content template. In particular, each trained machinelearning model corresponds to a machine learning model associated with acontent template that is trained using training data comprising traininginputs and training outputs.

At operation 430, the processing device, for each request, assigns therequest to a first group or a second group. In one implementation, theprocessing device can assign each request to either group based on aratio of a number of trained machine learning models (associated with acontent template) that satisfy the reliability criterion to a number oftrained machine learning models (associated with a content template)that do not satisfy the reliability criterion. For example, once theprocessing device determines the ratio, the processing device candetermine a number of requests to assign to the first group and a numberof requests to assign to the second group based on the ratio. In casethe ratio is 3:2, the processing device can assign 60% of the requeststo the first group and the rest (the 40% of the requests) to the secondgroup. Accordingly, the processing device can assign a request to thefirst group or the second until the determined quota for each group isreached.

In some embodiments, the processing device can assign a small portion,such as 5%, of the requests to another group and provide a randomlygenerated content item to the respective targets. The processing devicecan receive responses of the targets when such content items arepresented to the targets. The processing device can subsequently use theresponses to further (adaptively) train all machine learning modelsassociated with content templates and elements of the content templates.

At operation 440, the processing device, for each request in the firstgroup (the group of requests to be processed using sufficiently stabletrained machine learning models), performs the following operations 442to 446. At operation 442, the processing device inputs the respectiveset of characteristics associated with the request in the first groupinto each trained machine learning model in the first subset of thetrained machine learning models. In one implementation, the processingdevice can determine from the request the set of characteristics such asdemographic characteristics (such as age or gender), contextualcharacteristics (such as device brand, operating system, user time,geographic location), historical (or user behavioral) characteristics(such as a number of impressions, time since the last impression, numberof clicks).

At operation 444, the processing device selects a content template basedon outputs of the first subset of the trained machine learning models.In one implementation, in response to inputting the set ofcharacteristics to each machine learning model of the first subset, theprocessing device can obtain outputs from the machine learning models inthe first subset. Such an output can indicate a probability of thetarget having the set of characteristics performing a target action inresponse to being presented with a content item generated in accordancewith a content template associated with the respective trained machinelearning model. In addition, the output can be a numerical value between0 and 1—the higher the number, the higher the probability is. Based onthe outputs of the first subset of the trained machine learning models,the processing device can determine an output indicating a highestprobability of the respective target performing the target action.Accordingly, the processing device can identify a content templateassociated with the trained machine learning model that generated suchan output. The processing device can select the content template for therequested content item.

At operation 446, the processing device generates a content item basedon the selected content template. Details about how the content item isgenerated based on the selected content template is described below withrespect to FIGS. 5A-5B.

At operation 450, the processing device, for each request in the secondgroup, generates a content item based on a content template associatedwith one of the second subset of the trained machine learning models. Inone implementation, the processing device can determine a contenttemplate associated with each one of the second subset of the trainedmachine learning models. The processing device can determine any one ofthe content templates. In one implementation, the processing device canrandomly select one of the content templates. Subsequently, theprocessing device can generate the requested content item in accordancewith the determined content template.

For example, the templates 301 and 303 from FIGS. 3A and 3B can beassociated with the second subset of the trained machine learningmodels. The processing device can determine to use the template 301 inFIG. 3A to generate a content item, for example, an advertisement. Asillustrated, the template 301 has three components 302A to 302C—a title,an image, a call to action. Each component 302A-302C can be associatedwith one or more elements that can be presented as the respectivecomponent. For each component 302A-302C, the processing device canselect any one element from the associated elements. As illustrated inFIG. 3C, the processing device can, for example, generate the contentitem 350 using the first element for each component 302A-302C.

After the processing device generates the content item, the processingdevice can provide the content item to a content providing server thattransmitted a request for the content item. Subsequently, the contentproviding server can present the content item to a client device. Thecontent providing server may or may not receive a response from theclient device within a pre-determined time period. In one embodiment,the processing device can receive, for each content item generated foreach request in the second group, a respective indication as to whetherthe associated respective target performed a target action responsive topresentation of the respective content item to the respective target.The indication can contain binary information (e.g., yes or no, or 1 or0) as to whether the target action was performed or not. When receivingthe indication, the processing device can also receive metadataidentifying the request or the content item associated with theindication.

The processing device can train the second subset of the plurality oftrained machine learning models using the characteristics, the contentitems generated based on the characteristics, and the receivedindications associated with the content items. In one implementation,each trained machine learning model of the second subset of the trainedmachine learning models is trained for a respective content templateusing a) respective sets of characteristics of targets associated withrequests for which the respective content template was selected, b)respective content items generated for the respective requests and c)respective indications associated with the respective content items. Forexample, for each indication, the processing device can determine thecontent item associated with the indication and the set ofcharacteristics used in generating the content item. The processingdevice can determine a trained machine learning model from the secondsubset that is associated with the determined content item. Once theprocessing device identifies the trained machine learning modelassociated with the content item of the indication, the processingdevice can use the set of characteristics associated with the indicationas a training input and the indication as a label or target to furthertrain the machine learning model.

After the secondary training of the machine learning models in thesecond subset, the processing device can determine, for each trainedmachine learning model in the second subset, whether the trained machinelearning model satisfies the reliability criterion. Accordingly, theprocessing device can move those trained machine learning models fromthe second subset that satisfy the reliability criterion to the firstsubset. As such, while the stable or reliable models (i.e., machinelearning models in the first subset) reflecting strong signals orpatterns despite of being initially trained on limited data sets, areutilized to generate an optimized content item, the weak or lessreliable models can be further and adaptively trained based on moretraining data and data sets reflecting any changes in the pattern sincethe initial training. In this way, the present disclosure providesadaptive optimization of content item using continuous machine learningmodels.

In another embodiment, the processing device can receive indications oftarget action for every content item generated (e.g., including contentitems generated based on outputs of trained machine learning models inthe first subset). Then, the processing device can use the indicationsfor further training all the initially trained machine learning models(e.g., the machine learning models that are in the first and the secondsubset) that are associated with content templates. Accordingly, themachine learning models can be adjusted to further improve an accuracyof the machine learning models and/or to accommodate any changes in userbehavior since the initial training. Therefore, the machine learningmodels can be continuously trained to generate adaptively optimizedcontent items.

FIG. 5A depicts a flow diagram of a method 500 for generating a contentitem based on a selected content template, in accordance with someembodiments of the present disclosure. The method 500 may be performedby the content managing component 116 of FIG. 1 . The method 500 may beexecuted by one or more processing devices of the server 112. The method500 may be carried out for each content providing server 118A-118Z to bepresented to client devices 120A-120Z, 120ZA-120ZZ.

For simplicity of explanation, the method 500 of this disclosure isdepicted and described as a series of acts. However, acts in accordancewith this disclosure can occur in various orders and/or concurrently,and with other acts not presented and described herein. Furthermore, notall illustrated acts may be required to implement the method 500 inaccordance with the disclosed subject matter. In addition, those skilledin the art will understand and appreciate that the method 500 couldalternatively be represented as a series of interrelated states via astate diagram or events. Additionally, it should be appreciated that themethod 500 disclosed in this specification are capable of being storedon an article of manufacture (e.g., a computer program accessible fromany computer-readable device or storage media) to facilitatetransporting and transferring such method to computing devices.

At operation 510, the processing device determines components of aselected content template (e.g., the content template selected atoperation 444). As illustrated in FIGS. 3A to 3C, a content template mayinclude a plurality of components (e.g., a title, a call to action, animage, a video, an audio, a background image, a message, etc.). Eachcomponent can be associated with one or more elements. As shown in FIG.3C, the image component can have three different images (i.e.,elements), but only one image is to be included in a particular contentitem for the image component of the content template 301 of FIG. 3A.Hereinafter, the content template 301 of FIG. 3A will be used as anexample of the selected content template, for the purpose ofillustration.

At operation 520, the processing device, for each component of theselected content template associated with multiple elements, performsoperations 521 to 525. At operation 521, the processing deviceidentifies a first subset of the trained machine learning models(associated with elements of the respective component) that satisfy areliability criterion and a second subset of the trained machinelearning models (associated with elements of the respective component)that fail to satisfy the reliability criterion. In one implementation,each trained machine learning model is associated with a respectiveelement of the respective component of the selected content template.For example, referring back to the content item 350 of FIG. 3C, for theheadline component, there are four headline elements (i.e., fourdifferent headline messages). Thus, the set of trained machine learningmodels can correspond to the trained machine learning models for thefour headline elements. Each trained machine learning model can beassociated with each headline element.

Furthermore, each trained machine learning model is trained to output aprobability that a target associated with an input set ofcharacteristics would perform a target action responsive to beingpresented with a content item including the respective elementassociated with the respective trained machine learning model.Accordingly, each trained machine learning model can predict alikelihood or a probability of a target (e.g., a target audience of thecontent item) responding to a content item including the respectiveelement (e.g., the first headline in the content item 350 of FIG. 3C).Each trained machine learning model can be trained using training datacomprising training inputs and training outputs.

At operation 522, the processing device assigns the request to a thirdgroup or a fourth group. The processing device can perform operation 522for each component of the selected content template. For example, forthe headline component of FIG. 3C, the processing device can determinethe request should be assigned to one of the two groups (a group ofrequests to be processed using sufficiently reliable trained machinelearning models associated with elements of the headline component andthe other group of request to be processed by randomly orpseudo-randomly selecting one of elements (for the headline component)associated with less reliable trained machine learning models). For theimage component of FIG. 3C, the processing device can determine that therequest should be assigned to one of the two groups (a group of requeststo be processed using sufficiently reliable trained machine learningmodels associated with elements of the image component and the othergroup of request to be processed by randomly or pseudo-randomlyselecting one of elements (for the image component) associated with lessreliable trained machine learning models). The two groups for theheadline component should be different from the two groups for the imagecomponent.

Moreover, this request corresponds to one of the requests assigned tothe first group (to be processed using sufficiently reliable machinelearning models associated with content templates) at operation 440.Accordingly, in addition to being assigned to the first group, therequest can also be assigned to a third group or a fourth group. Similarto how requests were assigned to the first group or second group atoperation 430, the processing device can assign this request to thethird group or the fourth group based on a ratio of a number of trainedmachine learning models (associated with elements) that satisfy areliability criterion to a number of trained machine learning models(associated with elements) that does not satisfy the reliabilitycriterion.

For example, the processing device can determine the ratio to be 3:1. Byusing the headline component of the content item 350 in FIG. 3C, theprocessing device can determine that a trained machine learning modelfor the fourth headline does not satisfy the reliability criterion. Insuch a case, the processing device can assign 75% of the requests to thethird group and the rest (the 25% of the requests) to the fourth group.As another example, for the image component of the content item 350 inFIG. 3C, the processing device can determine that the ratio of a numberof trained machine learning models that satisfies a reliabilitycriterion to a number of trained machine learning models that do notsatisfy the reliability criterion for the image elements as 1:2. Inaddition, or in the alternative, the processing device can identify thatonly the trained machine learning model for the second image elementsatisfies the reliability criterion. Accordingly, the processing devicecan assign 33% of the requests to the third group and the rest to thefourth group.

At operation 523, the processing device, for the request assigned to thethird group, performs operations 524 to 525. At operation 524, theprocessing device inputs the respective set of characteristicsassociated with the request into each (that satisfy the reliabilitycriterion) of the first subset of the second plurality of trainedmachine learning models associated with elements for the respectivecomponent of the selected content template. The set of characteristicscan include demographic characteristics (such as age or gender),contextual characteristics (such as device brand, operating system, usertime, geographic location), historical (or user behavioral)characteristics (such as a number of impressions, time since the lastimpression, number of clicks) of the respective target (e.g., a targetaudience of the request).

At operation 525, the processing device selects an element to beincluded in the content item for the respective component of theselected content template, based on outputs of the first subset of thetrained machine learning models associated with elements. In oneimplementation, the processing device can select an element associatedwith a trained machine learning model that provided the highestprobability—the probability of the target having the set ofcharacteristics performing a target action in response to beingpresented with a content item including the element associated with therespective trained machine learning model in accordance with theselected content template.

At operation 526, for the request assigned to the fourth group, theprocessing device selects an element to be included in the content itemfor the respective component of the selected content template, based onan element associated with one of the second subset of the trainedmachine learning models associated with elements. Similar to operation450, the processing device can determine elements associated withtrained machine learning models included in the second subset (the weakor less stable machine learning models). The processing device canselect any one of the elements to be included in the content item inaccordance with the selected content template. After the operation 526,the processing device continues to operation 530 in FIG. 5B.

FIG. 5B depicts a flow diagram of a method 550 that is continued fromthe method of FIG. 5A for generating a content item based on a selectedcontent template, in accordance with some embodiments of the presentdisclosure. The method 550 may be performed by the content managingcomponent 116 of FIG. 1 . The method 550 may be executed by one or moreprocessing devices of the server 112. The method 550 may be carried outfor each content providing server 118A-118Z to be presented to clientdevices 120A-120Z, 120ZA-120ZZ.

At operation 530, the processing device, for each component of theselected content template having one element, selects the one element tobe included in the content item for the respective component of theselected content template.

At operation 540, the processing device generates the content item thatincludes the elements selected at operations 525, 526, and/or 530 inassociation with the components of the selected content template.

In some embodiments, the processing device can further train the machinelearning models in the second subset of the trained machine learningmodels associated with elements in a similar way as the secondarytraining of the machine learning models associated with contenttemplates as described above with respect to FIG. 4 . That is, for eachgenerated content item associated with each request in the fourth group(the group of requests to be provided with an element randomly selectedfrom elements associated with the trained machine learning models thatfails to satisfy the reliability criterion), the processing device canreceive an indication as to whether the associated respective targetperformed a target action responsive to presentation of the respectivecontent item including the selected element to the respective target.This information, along with information on the set of characteristicsof the target to which the content item was provided, may be used tofurther refine the appropriate trained machine learning model associatedwith the content item.

Subsequently, the processing device can train the second subset of thetrained machine learning models associated with elements using a) therespective set of characteristics of target associated with the requestfor which the respective element was selected, b) the respective contentitem including the associated respective element and c) the respectiveindication associated with the respective content item. After thesecondary training, the processing device can determine whether anytrained machine learning models can be moved from the second subset tothe first subset of trained machine learning models associated withelements based on the reliability criterion. In case the processingdevice determines one or more trained machine learning models that arenow sufficiently stable, reliable, or accurate, the processing devicecan move the trained machine learning models to the first subset.

FIG. 6 depicts a block diagram of an example computing system 600operating in accordance with one or more aspects of the presentdisclosure. In various illustrative examples, computer system 600 maycorrespond to any of the computing devices within system architecture100 of FIG. 1 . In one implementation, the computer system 600 may bethe content generating server 112.

In certain implementations, computer system 600 may be connected (e.g.,via a network, such as a Local Area Network (LAN), an intranet, anextranet, or the Internet) to other computer systems. Computer system600 may operate in the capacity of a server or a client computer in aclient-server environment, or as a peer computer in a peer-to-peer ordistributed network environment. Computer system 600 may be provided bya personal computer (PC), a tablet PC, a set-top box (STB), a PersonalDigital Assistant (PDA), a cellular telephone, a web appliance, aserver, a network router, switch or bridge, or any device capable ofexecuting a set of instructions (sequential or otherwise) that specifyactions to be taken by that device. Further, the term “computer” shallinclude any collection of computers that individually or jointly executea set (or multiple sets) of instructions to perform any one or more ofthe methods described herein.

In a further aspect, the computer system 600 may include a processingdevice 602, a volatile memory 604 (e.g., random access memory (RAM)), anon-volatile memory 606 (e.g., read-only memory (ROM) orelectrically-erasable programmable ROM (EEPROM)), and a data storagedevice 616, which may communicate with each other via a bus 608.

Processing device 602 may be provided by one or more processors such asa general purpose processor (such as, for example, a complex instructionset computing (CISC) microprocessor, a reduced instruction set computing(RISC) microprocessor, a very long instruction word (VLIW)microprocessor, a microprocessor implementing other types of instructionsets, or a microprocessor implementing a combination of types ofinstruction sets) or a specialized processor (such as, for example, anapplication specific integrated circuit (ASIC), a field programmablegate array (FPGA), a digital signal processor (DSP), or a networkprocessor).

Computer system 600 may further include a network interface device 622.Computer system 600 also may include a video display unit 610 (e.g., anLCD), an alphanumeric input device 612 (e.g., a keyboard), a cursorcontrol device 614 (e.g., a mouse), and a signal generation device 620.

Data storage device 616 may include a non-transitory computer-readablestorage medium 624 on which may store instructions 626 encoding any oneor more of the methods or functions described herein, includinginstructions implementing the content managing component 116 of FIG. 1for implementing the methods described herein.

Instructions 626 may also reside, completely or partially, withinvolatile memory 604 and/or within processing device 602 during executionthereof by computer system 600, hence, volatile memory 604 andprocessing device 602 may also constitute machine-readable storagemedia.

While computer-readable storage medium 624 is shown in the illustrativeexamples as a single medium, the term “computer-readable storage medium”shall include a single medium or multiple media (e.g., a centralized ordistributed database, and/or associated caches and servers) that storethe one or more sets of executable instructions. The term“computer-readable storage medium” shall also include any tangiblemedium that is capable of storing or encoding a set of instructions forexecution by a computer that cause the computer to perform any one ormore of the methods described herein. The term “computer-readablestorage medium” shall include, but not be limited to, solid-statememories, optical media, and magnetic media.

In the foregoing description, numerous details are set forth. It will beapparent, however, to one of ordinary skill in the art having thebenefit of this disclosure, that the present disclosure can be practicedwithout these specific details. In some instances, well-known structuresand devices are shown in block diagram form, rather than in detail, inorder to avoid obscuring the present disclosure.

Some portions of the detailed description have been presented in termsof algorithms and symbolic representations of operations on data bitswithin a computer memory. These algorithmic descriptions andrepresentations are the means used by those skilled in the dataprocessing arts to most effectively convey the substance of their workto others skilled in the art. An algorithm is here, and generally,conceived to be a self-consistent sequence of steps leading to a desiredresult. The steps are those requiring physical manipulations of physicalquantities. Usually, though not necessarily, these quantities take theform of electrical or magnetic signals capable of being stored,transferred, combined, compared, and otherwise manipulated. It hasproven convenient at times, for reasons of common usage, to refer tothese signals as bits, values, elements, symbols, characters, terms,numbers, or the like.

It should be borne in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise as apparent from the following discussion,it is appreciated that throughout the description, discussions utilizingterms such as “receiving”, “identifying”, “determining”, “generating”,“assigning”, “inputting”, “selecting”, “training”, “moving”, or thelike, refer to the actions and processes of a computer system, orsimilar electronic computing device, that manipulates and transformsdata represented as physical (e.g., electronic) quantities within thecomputer system's registers and memories into other data similarlyrepresented as physical quantities within the computer system memoriesor registers or other such information storage, transmission or displaydevices.

For simplicity of explanation, the methods are depicted and describedherein as a series of acts. However, acts in accordance with thisdisclosure can occur in various orders and/or concurrently, and withother acts not presented and described herein. Furthermore, not allillustrated acts can be required to implement the methods in accordancewith the disclosed subject matter. In addition, those skilled in the artwill understand and appreciate that the methods could alternatively berepresented as a series of interrelated states via a state diagram orevents. Additionally, it should be appreciated that the methodsdisclosed in this specification are capable of being stored on anarticle of manufacture to facilitate transporting and transferring suchmethods to computing devices. The term article of manufacture, as usedherein, is intended to encompass a computer program accessible from anycomputer-readable device or storage media.

Certain implementations of the present disclosure also relate to anapparatus for performing the operations herein. This apparatus can beconstructed for the intended purposes, or it can comprise a generalpurpose computer selectively activated or reconfigured by a computerprogram stored in the computer. Such a computer program can be stored ina computer readable storage medium, such as, but not limited to, anytype of disk including floppy disks, optical disks, CD-ROMs, andmagnetic-optical disks, read-only memories (ROMs), random accessmemories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, or any typeof media suitable for storing electronic instructions.

Reference throughout this specification to “one implementation” or “animplementation” means that a particular feature, structure, orcharacteristic described in connection with the implementation isincluded in at least one implementation. Thus, the appearances of thephrase “in one implementation” or “in an implementation” in variousplaces throughout this specification are not necessarily all referringto the same implementation. In addition, the term “or” is intended tomean an inclusive “or” rather than an exclusive “or.” Moreover, thewords “example” or “exemplary” are used herein to mean serving as anexample, instance, or illustration. Any aspect or design describedherein as “exemplary” is not necessarily to be construed as preferred oradvantageous over other aspects or designs. Rather, use of the words“example” or “exemplary” is intended to present concepts in a concretefashion.

It is to be understood that the above description is intended to beillustrative, and not restrictive. Many other implementations will beapparent to those of skill in the art upon reading and understanding theabove description. The scope of the disclosure should, therefore, bedetermined with reference to the appended claims, along with the fullscope of equivalents to which such claims are entitled.

What is claimed is:
 1. A system comprising: a memory; and a processingdevice, operatively coupled with the memory, the processing deviceconfigured to perform operations comprising: receiving a plurality ofrequests for content items, wherein each of the plurality of requests isassociated with a respective target comprising a respective set ofcharacteristics; identifying a first subset of a plurality of trainedmachine learning models that satisfy a reliability criterion and asecond subset of the plurality of trained machine learning models thatfail to satisfy the reliability criterion, wherein each trained machinelearning model of the plurality of trained machine learning models isassociated with a respective content template of a plurality of contenttemplates and is trained to output a probability that a targetassociated with an input set of characteristics would perform a targetaction responsive to being presented with a content item generated basedon the respective content template associated with the respectivetrained machine learning model; for each request of the plurality ofrequests, assigning the request to a first group or a second group basedon a ratio of a number of trained machine learning models in the firstsubset to a number of trained machine learning models in the secondsubset; and for each request in the first group, generating a contentitem based on a content template associated with one of the first subsetof the plurality of trained machine learning models.
 2. The system ofclaim 1, wherein generating the content item based on the contenttemplate associated with one of the first subset of the plurality oftrained machine learning models comprises: inputting the respective setof characteristics associated with the request into each of the firstsubset of the plurality of trained machine learning models; andselecting the content template based on outputs of the first subset ofthe plurality of trained machine learning models.
 3. The system of claim2, wherein selecting the content template comprises: determining, basedon the outputs of the first subset of the plurality of trained machinelearning models, the content template associated with an outputindicating a highest probability of the respective target performing thetarget action.
 4. The system of claim 1, wherein the operations furthercomprise: for each request in the second group, generating a secondcontent item based on a second content template associated with one ofthe second subset of the plurality of trained machine learning models.5. The system of claim 4, wherein generating the second content itembased on the second content template associated with one of the secondsubset of the plurality of trained machine learning models comprises:determining one of one or more content templates associated with thesecond subset of the plurality of trained machine learning models; andgenerating the second content item in accordance with the determinedcontent template.
 6. The system of claim 1, wherein the operationsfurther comprise: receiving, for each generated content item for eachrequest in the second group, a respective indication as to whether theassociated respective target performed a target action responsive topresentation of the respective content item to the respective target;training the second subset of the plurality of trained machine learningmodels, wherein each trained machine learning model of the second subsetof the plurality of trained machine learning models is trained for arespective content template of the plurality of content templates usinga) respective sets of characteristics of targets associated withrequests for which the respective content template was selected, b)respective content items generated for the respective requests and c)respective indications associated with the respective content items;determining, for each trained machine learning model in the secondsubset, whether the trained machine learning model satisfies thereliability criterion; and moving those trained machine learning modelsthat satisfy the reliability criterion from the second subset of theplurality of trained machine learning models to the first subset of theplurality of trained machine learning models.
 7. The system of claim 1,wherein generating the content item based on the content templatecomprises: determining a plurality of components of the contenttemplate, each component associated with one or more elements; for eachcomponent of the content template having a plurality of elements,performing the following comprising: identifying a first subset of asecond plurality of trained machine learning models that satisfy anotherreliability criterion and a second subset of the second plurality oftrained machine learning models that fail to satisfy the anotherreliability criterion, wherein each trained machine learning model ofthe second plurality of trained machine learning models is associatedwith a respective element of a respective component of the contenttemplate and is trained to output a probability that a target associatedwith an input set of characteristics would perform a target actionresponsive to being presented with a content item including therespective element associated with the respective trained machinelearning model of the second plurality of trained machine learningmodels; assigning the request to a third group or a fourth group; foreach request in the third group, performing the following comprising:inputting the respective set of characteristics associated with therequest into each of the first subset of the second plurality of trainedmachine learning models; and selecting an element to be included in thecontent item for the respective component of the content template, basedon outputs of the first subset of the second plurality of trainedmachine learning models; and for each request in the fourth group,selecting an element to be included in the content item for therespective component of the content template, based on an elementassociated with one of the second subset of the second plurality oftrained machine learning models; for each component of the contenttemplate having one element, selecting the one element to be included inthe content item for the respective component of the content template;and generating a content item comprising the selected elementsassociated with the plurality of components of the content template. 8.The system of claim 7, wherein a trained machine learning model of thesecond plurality of trained machine learning models corresponds to amachine learning model associated with an element associated with acomponent of the content template that is trained using training datacomprising training inputs and training outputs, the training inputsincluding a plurality set of characteristics associated with targets,and the training outputs including a plurality of indications as towhether corresponding target actions were performed by the targetsresponsive to being presented with content items comprising therespective element.
 9. The system of claim 7, wherein assigning therequest to the third group or the fourth group comprises: determining asecond ratio of a number of trained machine learning models in the firstsubset of the second plurality of trained machine learning models to anumber of trained machine learning models the second subset of thesecond plurality of trained machine learning models; determining anumber of requests to assign to the third group and another number ofrequests to assign to the fourth group based on the second ratio; andassigning the request to the third group or the fourth group based onthe number of requests to assign to the third group and the anothernumber of requests to assign to the fourth group.
 10. The system ofclaim 7, wherein the operations further comprise: receiving, for eachgenerated content item for each request in the fourth group, arespective indication as to whether the associated respective targetperformed a target action responsive to presentation of the respectivecontent item to the respective target; training the second subset of thesecond plurality of trained machine learning models, wherein eachtrained machine learning model of the second subset of the secondplurality of trained machine learning models is trained for a respectiveelement of a component of the content template using a) respective setsof characteristics of targets associated with requests for which therespective element was selected, b) respective content items includingthe associated respective element and c) respective indicationsassociated with the respective content items; determining, for eachtrained machine learning model in the second subset of the secondplurality of trained machine learning models, whether the trainedmachine learning model satisfies the reliability criterion; and movingthose trained machine learning models that satisfy the anotherreliability criterion from the second subset of the second plurality oftrained machine learning models to the first subset of the secondplurality of trained machine learning models.
 11. The system of claim10, wherein the another reliability criterion is associated with atleast one of a mean or a standard deviation of an area under receiveroperating characteristic (ROC) curve determined for a respective trainedmachine learning model of the second plurality of trained machinelearning models using one or more sets of validation data, thevalidation data comprising validation input data and validation outputdata, wherein the validation input data includes another set ofcharacteristics associated with another target and the validation outputdata includes an indication of whether or not another target action wasperformed.
 12. The system of claim 1, wherein the reliability criterionis associated with at least one of a mean or a standard deviation of anarea under receiver operating characteristic (ROC) curve determined fora respective trained machine learning model of the plurality of trainedmachine learning models using one or more sets of validation data, thevalidation data comprising validation input data and validation outputdata, wherein the validation input data includes another set ofcharacteristics associated with another target and the validation outputdata includes an indication of whether or not another target action wasperformed.
 13. The system of claim 1, wherein assigning the request tothe first group or the second group comprises: determining the ratio ofthe number of trained machine learning models in the first group to thenumber of trained machine learning models the second group; determininga number of requests to assign to the first group and another number ofrequests to assign to the second group based on the ratio; and assigningthe request to the first group or the second group based on the numberof requests to assign to the first group and the another number ofrequests to assign to the second group.
 14. The system of claim 1,wherein a trained machine learning model of the plurality of trainedmachine learning models corresponds to a machine learning model that isassociated with the content template and that is trained using trainingdata comprising training inputs and training outputs, the traininginputs including a plurality set of characteristics associated withtargets, and the training outputs including a plurality of indicationsas to whether corresponding target actions were performed by the targetsresponsive to being presented with content items based on the contenttemplate.
 15. A method comprising: receiving a plurality of requests forcontent items, wherein each of the plurality of requests is associatedwith a respective target comprising a respective set of characteristics;identifying a first subset of a plurality of trained machine learningmodels that satisfy a reliability criterion and a second subset of theplurality of trained machine learning models that fail to satisfy thereliability criterion, wherein each trained machine learning model ofthe plurality of trained machine learning models is associated with arespective content template of a plurality of content templates and istrained to output a probability that a target associated with an inputset of characteristics would perform a target action responsive to beingpresented with a content item generated based on the respective contenttemplate associated with the respective trained machine learning model;for each request of the plurality of requests, assigning the request toa first group or a second group based on a ratio of a number of trainedmachine learning models in the first subset to a number of trainedmachine learning models in the second subset; and for each request inthe first group, generating a content item based on a content templateassociated with one of the first subset of the plurality of trainedmachine learning models.
 16. The method of claim 15, wherein generatingthe content item based on the content template associated with one ofthe first subset of the plurality of trained machine learning modelscomprises: inputting the respective set of characteristics associatedwith the request into each of the first subset of the plurality oftrained machine learning models; selecting the content template based onoutputs of the first subset of the plurality of trained machine learningmodels, wherein the content template is associated with an outputindicating a highest probability of the respective target performing thetarget action; and generating the content item based on the contenttemplate by performing the following comprising: determining a pluralityof components of the content template, each component associated withone or more elements; for each component of the content template havinga plurality of elements, performing the following comprising:identifying a first subset of a second plurality of trained machinelearning models that satisfy another reliability criterion and a secondsubset of the second plurality of trained machine learning models thatfail to satisfy the another reliability criterion, wherein each trainedmachine learning model of the second plurality of trained machinelearning models is associated with a respective element of a respectivecomponent of the content template and is trained to output a probabilitythat a target associated with an input set of characteristics wouldperform a target action responsive to being presented with a contentitem including the respective element associated with the respectivetrained machine learning model of the second plurality of trainedmachine learning models; assigning the request to a third group or afourth group; for each request in the third group, performing thefollowing comprising: inputting the respective set of characteristicsassociated with the request into each of the first subset of the secondplurality of trained machine learning models; and selecting an elementto be included in the content item for the respective component of thecontent template, based on outputs of the first subset of the secondplurality of trained machine learning models; and for each request inthe fourth group, selecting an element to be included in the contentitem for the respective component of the content template, based on anelement associated with one of the second subset of the second pluralityof trained machine learning models; for each component of the contenttemplate having one element, selecting the one element to be included inthe content item for the respective component of the content template;and generating a content item comprising the selected elementsassociated with the plurality of components of the content template. 17.The method of claim 15, further comprising: for each request in thesecond group, generating a second content item based on a second contenttemplate associated with one of the second subset of the plurality oftrained machine learning models.
 18. The method of claim 15, furthercomprising: receiving, for each generated content item for each requestin the second group, a respective indication as to whether theassociated respective target performed a target action responsive topresentation of the respective content item to the respective target;training the second subset of the plurality of trained machine learningmodels, wherein each trained machine learning model of the second subsetof the plurality of trained machine learning models is trained for arespective content template of the plurality of content templates usinga) respective sets of characteristics of targets associated withrequests for which the respective content template was selected, b)respective content items generated for the respective requests and c)respective indications associated with the respective content items;determining, for each trained machine learning model in the secondsubset, whether the trained machine learning model satisfies thereliability criterion; and moving those trained machine learning modelsthat satisfy the reliability criterion from the second subset of theplurality of trained machine learning models to the first subset of theplurality of trained machine learning models.
 19. A non-transitorycomputer readable storage medium comprising instructions that, whenexecuted by a processor, cause the processor to perform operationscomprising: receiving a plurality of requests for content items, whereineach of the plurality of requests is associated with a respective targetcomprising a respective set of characteristics; identifying a firstsubset of a plurality of trained machine learning models that satisfy areliability criterion and a second subset of the plurality of trainedmachine learning models that fail to satisfy the reliability criterion,wherein each trained machine learning model of the plurality of trainedmachine learning models is associated with a respective content templateof a plurality of content templates and is trained to output aprobability that a target associated with an input set ofcharacteristics would perform a target action responsive to beingpresented with a content item generated based on the respective contenttemplate associated with the respective trained machine learning model;for each request of the plurality of requests, assigning the request toa first group or a second group based on a ratio of a number of trainedmachine learning models in the first subset to a number of trainedmachine learning models in the second subset; and for each request inthe first group, generating a content item based on a content templateassociated with one of the first subset of the plurality of trainedmachine learning models.
 20. The non-transitory computer readablestorage medium of claim 19, wherein generating the content item based onthe content template associated with one of the first subset of theplurality of trained machine learning models comprises: inputting therespective set of characteristics associated with the request into eachof the first subset of the plurality of trained machine learning models;and selecting the content template based on outputs of the first subsetof the plurality of trained machine learning models, wherein the contenttemplate is associated with an output indicating a highest probabilityof the respective target performing the target action.
 21. Thenon-transitory computer readable storage medium of claim 19, furthercomprising: determining one of one or more content templates associatedwith the second subset of the plurality of trained machine learningmodels; and for each request in the second group, generating a secondcontent item based on the determined content template.
 22. Thenon-transitory computer readable storage medium of claim 19, furthercomprising: receiving, for each generated content item for each requestin the second group, a respective indication as to whether theassociated respective target performed a target action responsive topresentation of the respective content item to the respective target;training the second subset of the plurality of trained machine learningmodels, wherein each trained machine learning model of the second subsetof the plurality of trained machine learning models is trained for arespective content template of the plurality of content templates usinga) respective sets of characteristics of targets associated withrequests for which the respective content template was selected, b)respective content items generated for the respective requests and c)respective indications associated with the respective content items;determining, for each trained machine learning model in the secondsubset, whether the trained machine learning model satisfies thereliability criterion; and moving those trained machine learning modelsthat satisfy the reliability criterion from the second subset of theplurality of trained machine learning models to the first subset of theplurality of trained machine learning models.
 23. The non-transitorycomputer readable storage medium of claim 19, wherein generating thecontent item based on the content template further comprises:determining a plurality of components of the content template, eachcomponent associated with one or more elements; for each component ofthe content template having a plurality of elements, performing thefollowing comprising: identifying a first subset of a second pluralityof trained machine learning models that satisfy another reliabilitycriterion and a second subset of the second plurality of trained machinelearning models that fail to satisfy the another reliability criterion,wherein each trained machine learning model of the second plurality oftrained machine learning models is associated with a respective elementof a respective component of the content template and is trained tooutput a probability that a target associated with an input set ofcharacteristics would perform a target action responsive to beingpresented with a content item including the respective elementassociated with the respective trained machine learning model of thesecond plurality of trained machine learning models; assigning therequest to a third group or a fourth group; for each request in thethird group, performing the following comprising: inputting therespective set of characteristics associated with the request into eachof the first subset of the second plurality of trained machine learningmodels; and selecting an element to be included in the content item forthe respective component of the content template, based on outputs ofthe first subset of the second plurality of trained machine learningmodels; and for each request in the fourth group, selecting an elementto be included in the content item for the respective component of thecontent template, based on an element associated with one of the secondsubset of the second plurality of trained machine learning models; foreach component of the content template having one element, selecting theone element to be included in the content item for the respectivecomponent of the content template; and generating a content itemcomprising the selected elements associated with the plurality ofcomponents of the content template.
 24. The non-transitory computerreadable storage medium of claim 19, wherein a trained machine learningmodel of the plurality of trained machine learning models corresponds toa machine learning model that is associated with the content templateand that is trained using training data comprising training inputs andtraining outputs, the training inputs including a plurality set ofcharacteristics associated with targets, and the training outputsincluding a plurality of indications as to whether corresponding targetactions were performed by the targets responsive to being presented withcontent items based on the content template.
 25. The non-transitorycomputer readable storage medium of claim 19, wherein the reliabilitycriterion is associated with at least one of a mean or a standarddeviation of an area under receiver operating characteristic (ROC) curvedetermined for a respective trained machine learning model of theplurality of trained machine learning models using one or more sets ofvalidation data, the validation data comprising validation input dataand validation output data, wherein the validation input data includesanother set of characteristics associated with another target and thevalidation output data includes an indication of whether or not anothertarget action was performed.
 26. The non-transitory computer readablestorage medium of claim 19, wherein assigning the request to the firstgroup or the second group comprises: determining the ratio of the numberof trained machine learning models in the first group to the number oftrained machine learning models the second group; determining a numberof requests to assign to the first group and another number of requeststo assign to the second group based on the ratio; and assigning therequest to the first group or the second group based on the number ofrequests to assign to the first group and the another number of requeststo assign to the second group.